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WWC Study Ratings 3 
Baker (1997) 


Causal Validity: 


Meets WWC Evidence Standards, a Randomized Controlled Trial with No 
Randomization, Attrition, or Disruption Problems 


Participants were randomly assigned to the intervention and control groups. There was a significant difference between 
groups on the Objective by Strands pretest, but this was addressed by the study author. There was severe overall attrition, but 
it did not affect the initial equivalence of the groups analyzed. Other than the pretest, no extraneous events were identified 
that appeared to be confounded with the intervention’s effect. 


Other Study 
Characteristics 


Study Rating 


Study-Specific Information 


Intervention Fidelity 


•• 


The intervention is well defined and implemented and meets the definition of Middle- 
School Math. The intervention was a year-long 8th-grade math curriculum that 
included 196 hours of computer-assisted lessons that were delivered by a regular 
classroom teacher. The intervention curriculum contained 196 lessons that were 
presented every other day for 85 minutes. 


Outcome Measures 


•• 


The test appeared to be appropriately aligned, but the reliability calculations are based 
on 62 survey items, although it was reported that the survey contained 78 items. The 
outcome measure used in the study measured the content that Middle-School Math is 
meant to affect. The primary outcome measure was the Objectives by Strands test 
(Mathematical Concepts and Applications Survey). 


People, Settings, and 
Timing 


• 


The number of participants and settings was too small and homogeneous to allow 
generalization to the full range of people and settings that are the target of Middle- 
School Math interventions. The study sample included 90 8th-graders taught by one 
math teacher in four classrooms — two intervention classes and two control classes at a 
middle school in suburban Missouri. The Objectives by Strands test was administered 
at the end of the study. 


Testing within 
Subgroups 


• 


The intervention effect was tested across the entire sample but not within important 
variations in settings. 


Analysis 


• 


The results were analyzed at the level of individual students, which matches the unit of 
randomization. In this study, natural student groupings (classrooms, schools, etc.) may 
have affected findings. Although the author’s analysis does not address this grouping 
problem, the author does not report significant positive findings, so the impact of 
groupings on findings is likely minimal. The sample size was too small to allow for a 
precise estimate of effect. 


Statistical Reporting 


•• 


The author reported unadjusted means and standard deviations of the outcome measure, 
Objectives by Strands test, as well as the sample sizes for both the intervention and 
control groups, so an estimate of its effect could be calculated. A one-way analysis of 
covariance was performed to compare the Objectives by Strands scores of the 
intervention and control groups. The mean square and F statistic for comparison of the 
study groups on the Objectives by Strands test were also reported. 


Summary of Results. There was no statistically significant difference in student performance between students receiving 
the Expert Mathematician curriculum and those in the control groups. Baker reported negligible differences between the 
Expert Mathematician intervention and control groups. The outcome measure is sufficiently reliable, based on the author’s 
report of its reliability, but the sample size was relatively small. In addition, the sampled participants and settings were 
restricted. 


Note. • • Fully meets criteria; • Meets minimum criteria; X Does not meet criteria. 
a For more information on the criteria used to rate this studv, see the WWC Studv Review Standards. 
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Intervention: The Expert 
Mathematician, Version 3.0 

Operational Features 

The Expert Mathematician (version 3.0), the 
treatment curriculum, was developed by the 
author, J. J. Baker, from LogoWriter. Logo Writer 
is a computer program that the developer claims is 
intended to help children learn independently 
through exploration. It is designed to provide a 
creative environment that will help foster a child’s 
higher learning s ki lls. 

The lessons are based on Papert’s constructionist 
theory. This theory posits that children learn 
better if they are able to build on their prior 
knowledge, which provides meaning as they 
construct new knowledge in situ. The operational 
difference between Papert’s constructionism and 
Piaget’s constructivism is that, to facilitate 
learning, Papert takes into account the social 
aspects of learning and the child’s culture. 

The curriculum package contains 196 lessons, 
each intended to take about 40-120 minutes. 

Baker describes the curriculum as covering the 
range of concepts and content areas in the 
National Council of Teachers of Mathematics 
(NCTM) standards. The content areas covered in 
the study included arithmetic and geometry. 

According to Baker, in a typical Expert 
Mathematician class period, the teacher 
introduces the lesson for the day by using 
printed materials. Students then work 
individually or in pairs (using the printed 
materials and the computer) to study a new 
procedure to solve a math problem. 

The author randomly assigned students to 
intervention and control groups. The 
intervention groups were taught using the 
Expert Mathematician curriculum. 

People, Settings, and Timing 

The participants were 8th-grade students from a 
suburban middle school in St. Louis, Missouri. 
Most of the students were from low-income 
families and qualified for free or reduced-price 
lunch. All but three students were white. Baker 



reported that the students varied in their ability 
and none were in special education. 

Cost Information 

Cost information was unavailable for the Expert 
Mathematician curriculum used in this study. 

Intended Duration 

Baker did not explicitly report on the length of 
the intervention curriculum but stated that it 
contained 196 lessons. A class period was 85 
minutes in length, and classes were held on 
alternate days. During one class period, between 
one and two lessons could be completed. 

Scientific Rationale 

Baker developed the treatment curriculum, 
which he based in part on Papert’s theory that 
children understand mathematical concepts 
better when they can manipulate real objects 
rather than having to learn abstract concepts. He 
theorized that students would learn math better 
within a culturally relevant and individually 
tailored context, because meaning can be 
derived only within a context. Papert sees the 
LogoWriter environment as one that facilitates 
learning. Thus, the author’s choice to use the 
LogoWriter to develop a computer application 
for the curriculum was influenced by the fact 
that it is a software program that facilitates 
learning in a constructionist context. He 
compared his curriculum with the traditional, 
teacher-directed curriculum. 

Overview of the Study 

Purpose 

The main purpose of the study was to determine 
the relative effectiveness of a constructionist, 
computer-based curriculum (The Expert 
Mathematician [which Baker called “generative 
curriculum”]) compared with a teacher-directed 
curriculum (Transition Mathematics [which he 
called “linear curriculum”]). Furthermore, Baker 
tested for a relationship between students’ math 
attitudes and their math achievement as well as 
possible gender differences in this relationship. 
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Intervention Fidelity 

Class periods were 85 minutes each, on alternate 
days. Baker checked in with the teacher 
frequently over the course of the school year, 
which was the duration of this study. 

In the intervention groups, as students walked 
into the class, they had to pick up their manila 
folders, which contained the printed lesson for 
the day. The teacher checked their homework 
assignment before discussing the day’s lesson. 
After the introduction, the students were 
instructed to work in pairs at the computer. 

Homework assignments were provided either on 
paper or on the overhead. The students were 
allowed to start on their homework whenever 
they were done with the day’s unit, which was 
usually made up of one or two lessons. Baker 
reported that the students were not consistent in 
completing their homework and that the teacher 
was not consistent in assigning it. To keep the 
students on task during class, the teacher walked 
around the class while the students worked at 
their computers. No training of the teacher was 
reported. 

Outcome Measures 
One of the outcome measures was the 
Objectives by Strands test (sometimes referred 
to by Baker as the “Mathematical Concepts and 
Applications Survey”), which was used to 
measure the students’ mathematics achievement 
before and after the intervention. Baker noted 
that the test was developed by a large urban 
school district and contained 78 multiple-choice 
items. 

The other outcome measure was the Fennema- 
Sherman Mathematics Attitude Survey, which 
tapped a student’s experiences in learning math. 
Baker used it to develop eight attitude 
dimensions for this study. 

In WWC reports, only math achievement 
outcome measures are evaluated. Hence, the 
focus for the rest of this report is on the data 
from the Objectives by Strands test. 

Both the intervention and control students were 
pretested on these two outcome measures in the 



early fall. They were retested on these measures 
during the last week of the school year. 

Research Design 

The research design for this study was two- 
group comparison with random assignment. At 
the start of the study, a computer algorithm was 
used to divide at random a total of 90 students 
into two groups — intervention or control. Then 
the students were again divided randomly, with 
a flip of the coin, into four classes — two 
intervention classes and two control classes. 
Baker did not provide further detail on the 
methods used to generate or implement the 
random assignment sequence, such as who was 
responsible for generating the sequence, what 
recruitment methods were used, who was 
responsible for enrolling and assigning 
participants, or whether participants were blind 
to group assignment. 

Baker reported that, on average, the size of each 
of the four classes was about 23 students. The 
same teacher taught all four classes. The unit of 
assignment (student) matched the unit of 
analysis. 

Participant Flow 

At the beginning of the study, Baker reported a 
total of 90 students, with each of the four classes 
in the study having an average of 23 students. 

By the end of the study, during the analysis, 
only 70 students had both pretest and posttest 
scores. Of the 70 students, 36 were in the 
intervention group and 34 were in the control 
group. (See Figure 1, Participant Flow.) 

Reference Periods 

Baker did not provide reference periods for 
when recruitment, baseline data collection, 
intervention implementation, or outcome 
measurement occurred. 
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Baseline Data 

Baker did not describe baseline characteristics 
by intervention and control groups. He provided 
information on the pretest scores solely for the 



students who participated in both the pretest and 
posttest. Pretest scores were not provided for the 
students who dropped out between the pretest 
and posttest. (See table 1.) 



Figure 1. Participant Flow 



Random 

Allocation 



Intervention Allocation 



Purposeful Exclusions 
from Analysis: 

0 students excluded from 
analysis 



Lost to Analysis: 
20 students lost 
to analysis 



Analysis Sample 




Table 1. Pretest Characteristics of the Study Sample 





Intervention 

group 


Control 

group 


Intervention- 
control 
p value 


Pretest on math achievement: 








Objectives by Strands 








Mean ( SD ) 


28.1 (12.90) 


33.4 (11.50) 


<.05 


Sample size 


36 


34 





Statistical Methods 

First, Baker did a two-way analysis of variance 
(ANOVA) with condition and gender as the two 
variables. However, the intervention and control 
groups differed on the pretest math achievement 
score, with the intervention group scoring 
significantly lower than the control group. 
Hence, a two-way analysis of covariance 



(ANCOVA) was done with the Objectives by 
Strands pretest as the covariate. This same test 
was used as the posttest. 

Outcomes and Estimation 

Table 2 and Figure 2 present the effects of The 
Expert Mathematician on mathematical 
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achievement, as reported by Baker. In one of 
several analyses, Baker (1997) found that there 
was no significant difference in gain scores 
between students in the Expert Mathematician 
and the control groups when controlling on 
pretest. However, Baker (1997) did report that 
the average gain in scores of students in the 
Expert Mathematician group were similar to the 
gain scores of students in the control group on 
the Objectives by Strands test. In this study, 
natural student groupings (classrooms, schools, 
etc.) may have affected findings. Although the 
author’s analysis does not address this grouping 



problem, the author does not report significant 
positive findings, so the impact of groupings on 
findings is likely minimal. The WWC has not 
compared effect sizes for this study report 
because the possible effects of grouping on 
findings must first be addressed. 

The author also reports the effects of the 
curriculum on the students’ attitudes toward 
mathematics; however, this report focuses on 
the achievement results only. Baker does report 
findings by gender but does not separate them 
by intervention and control, and therefore they 
are not presented. 



Table 2. Impact Reported by Baker 





Intervention 
group posttest 


Control 
group posttest 


Intervention group 
estimated effect size 
(and significance) 


Posttest on math achievement: 
Objectives by Strands 
Mean (SD) 

Sample size 


39.8 (12.03) 
36 


40.8 (12.41) 
34 


NR a 



a The author did not report these effect sizes or the significance of the difference between posttest scores. 



Figure 2. Impact Reported by Baker 3 
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a Confidence intervals were computed by the WWC. 

b The intervention group scores were not significantly lower than the control group scores (p < .05). 
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WWC Evidence Criteria: Middle School Math 3 

Causal Validity 

The WWC evidence criteria for determining the level of evidence of a study reviewed under the topic Middle 
School Math Interventions are: 

Meets Evidence Standards 

■ Randomized controlled trial with no randomization, b attrition, or disruption problems 

■ Regression discontinuity study with no comparability, attrition, or disruption problems 

Meets Evidence Standards with Reservations 

■ Randomized controlled trial with a randomization, 0 attrition, and/or disruption problem 

■ Regression discontinuity study with a comparability, attrition, or disruption problem 

■ Quasi-experimental design with equivalent groups and no problems with attrition or disruption 

Other Study Characteristics 

In addition to determining whether a study Meets Evidence Standards or Meets Evidence Standards with Reservations, 
the WWC also assesses the strength of a study’s evidence based on the following other study characteristics: 
Intervention Fidelity. A study fully meets criteria for Intervention Fidelity (••) if the intervention contains most 
of the key characteristics that commonly define it, the author provides evidence of good implementation, and the 
intervention is documented well enough for others to replicate it. A study meets the minimum criteria (•) if the 
author does not evaluate implementation or finds partial implementation, or the intervention is not documented. A 
study is excluded from the review (X) if it does not meet the initial screening requirements for the intervention by 
omitting key characteristics of Middle School Math. 

Outcome Measures. A study fully meets criteria for Outcome Measures (••) if the outcome measure has face 
validity and reliability, and is not too closely aligned d to the content of the intervention. A study meets the 
minimum criteria (•) if the outcome measure is not too closely aligned to the content of the intervention. A study 
is excluded from the review (X) if it does not meet initial screening requirements by not focusing on important 
Middle School Math outcomes or if it lacks face validity and/or reliability. 

People, Settings, and Timing. A study fully meets criteria for People, Settings, and Timing (••) if it broadly 
samples from the people (units of interest) and settings that are the target of the intervention and the outcomes are 
measured at an appropriate time. A study meets the minimum criteria (•) if narrow but relevant samples and 
settings are included. A study is excluded from the review (X) if it does not include at least a relevant narrow 
sample of people or settings. 

Testing within Subgroups. A study fully meets criteria for Testing within Subgroups (••) if it identifies important 
subgroups among its sample and settings, and tests the intervention effect within each subgroup separately. A 
study meets the minimum criteria (•) if it simply tests the intervention effect across the entire sample. A study is 
not excluded from the review based on this standard. 

Analysis. A study fully meets criteria for Analysis (••) if the analysis is conducted at the same level (for 
example, students, classes, schools) as the unit of assignment and the unit of intervention delivery or if there is a 
mismatch between units but sufficient information is provided to permit an approximate estimation of the 
intervention’s effect and in either case, the data characteristics support the analysis. The study meets the minimum 
criteria (•) if an approximate estimation of effect at the level of assignment cannot be made. A study is not 
excluded from the review based on this standard. 

Statistical Reporting. A study fully meets criteria for Statistical Reporting (••) if the findings are reported for 
most outcome measures and effect sizes can be calculated. The study meets the minimum criteria (•) if findings 
are reported and effect sizes can be calculated for only some outcome measures. A study is excluded from the 
review (X) if it does not report findings for any relevant outcome measures. 

Note. For each study characteristic, the WWC considers a number of features to determine if the study fully meets criteria of that 
characteristic (••), meets minimum criteria (•), or does not meet minimum criteria (X). 

a These criteria are applied to studies that have passed initial WWC screening for Middle School Math. For more information on screening 
requirements . 

b Studies with randomization problems that make statistical adjustments Meet Evidence Standards. 

c Studies with randomization problems that do not make statistical adjustments Meet Evidence Standards with Reservations. 
d An overaligned outcome measure uses material that was part of the intervention. The control group was not exposed to this material. 
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Intervention Developer Contact 
Information 

The Expert Mathematician at: 
www.expertmath.org : 
email: frstprin@mninter.net : 
telephone: (612) 872-6741. 

Related Studies 

To see reports on other studies of the 
Expert Mathematician . 

How Can You Find Out More? 

To learn more about this study, read the 
original study (PDF) 



Report Production 

Date created: June 30, 2004 

Topic area reviewed under: Curriculum-Based 
Interventions for Increasing K-12 Math 
Achievement — Middle School. 
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